Using machine-learning models to determine movements of a mouth corresponding to live speech

ABSTRACT

Disclosed systems and methods predict visemes from an audio sequence. A viseme-generation application accesses a first set of training data that includes a first audio sequence representing a sentence spoken by a first speaker and a sequence of visemes. Each viseme is mapped to a respective audio sample of the first audio sequence. The viseme-generation application creates a second set of training data adjusting a second audio sequence spoken by a second speaker speaking the sentence such that the second and first sequences have the same length and at least one phoneme occurs at the same time stamp in the first sequence and in the second sequence. The viseme-generation application maps the sequence of visemes to the second audio sequence and trains a viseme prediction model to predict a sequence of visemes from an audio sequence.

TECHNICAL FIELD

This disclosure relates generally to animating virtual characters. Morespecifically, but not by way of limitation, this disclosure relates tousing machine-learning models to determine an appearance of an animatedmouth based on a sequence of speech samples.

BACKGROUND

Animation of virtual characters is a popular storytelling medium acrossmany domains. But traditional workflows for doing so are laborintensive. For example, animators often draw every frame by hand, ormanually specify how characters move when uttering a particular word.Animators specify how a character's lips move in accordance with thecharacter's speech. For example, when a character utters the syllable“a,” the character's mouth makes the same shape that a human's mouthwould make when speaking the syllable.

Automated animation removes the burden of hand-animating every mouthmovement. For example, in live or performance animation, a computingsystem controls cartoon characters in response to an animator's input orspeech. But existing solutions either cannot operate in real time, i.e.,perform live animation, or are not able to provide an animation that isrealistic and accurate. For example, existing solutions can result in acharacter's mouth not moving at all or moving too much relative to anexpected movement.

Additionally, solutions for live animation are often based on predictionmodels that predict animation sequences from speech. But such modelsrequire the use of training data, which is time-consuming to generatebecause audio sequences are hand-mapped to visemes. One minute of speechcan take five to seven hours of work to hand-animate.

Accordingly, improved solutions are needed for live animation andgenerating training data for prediction models that are used for liveanimation.

SUMMARY

Systems and methods are disclosed herein for predicting visemes from anaudio sequence. In an example, a viseme-generation application accessesa first set of training data. The first set of training data includes afirst audio sequence representing a sentence spoken by a first speaker,having a first length, and representing a sequence of phonemes and asequence of visemes. Each viseme is mapped to a respective audio sampleof the first audio sequence. The viseme-generation application creates asecond set of training data by accessing a second audio sequencerepresenting the sentence spoken by a second speaker, having a secondlength, and including the sequence of phonemes. The viseme-generationapplication adjusts the second audio sequence such that the secondsequence length is equal to the first length and at least one phonemeoccurs at the same time stamp in the first sequence and in the secondsequence. The viseme-generation application maps the sequence of visemesto the second audio sequence. The viseme-generation application trains aviseme prediction model to predict a sequence of visemes from an audiosequence.

These illustrative embodiments are mentioned not to limit or define thedisclosure, but to provide examples to aid understanding thereof.Additional embodiments are discussed in the Detailed Description, andfurther description is provided there.

BRIEF DESCRIPTION OF THE FIGURES

Features, embodiments, and advantages of the present disclosure arebetter understood when the following Detailed Description is read withreference to the accompanying drawings.

FIG. 1 is a diagram depicting a viseme-generation system, according tocertain embodiments of the present disclosure.

FIG. 2 depicts an example of a viseme set used by a viseme-generationsystem, according to certain embodiments of the present disclosure.

FIG. 3 is a flowchart that depicts an example of a process forgenerating a sequence of visemes from an audio sequence, according tocertain embodiments of the present disclosure.

FIG. 4 depicts an example of feature vector used by a viseme-generationsystem, according to certain embodiments of the present disclosure.

FIG. 5 depicts an example of a LSTM neural network used byviseme-generation application, according to certain embodiments of thepresent disclosure.

FIG. 6 is a flowchart that depicts an example of a process for traininga viseme-generation system, according to certain embodiments of thepresent disclosure.

FIG. 7 is a flowchart that depicts an example of a process forgenerating training data, according to certain embodiments of thepresent disclosure.

FIG. 8 depicts an example of time-warping used to generate trainingdata, according to certain embodiments of the present disclosure.

FIG. 9 depicts an example of a computing system for implementing certainembodiments of the present disclosure.

DETAILED DESCRIPTION

Embodiments described herein use time-warping techniques to automate thegeneration of robust and diverse training data sets used to trainpredictive models used in live and performance animation systems, and insome cases, apply these models to automate animation based on an audiosequence. As discussed above, generating training data for predictivemodels used in animation systems is cumbersome and time consuming.

In an example, viseme-generation application accesses a first set oftraining data. The first set of training data includes a first audiosequence and a sequence of visemes that can be used to animate acharacter. The first audio sequence represents a sequence of phonemes,or sounds, from a sentence spoken by a first speaker. Each viseme in thesequence of visemes corresponds to a respective audio sample in thefirst audio sequence. For example, a viseme has a time stampcorresponding to a time at which the first speaker uttered a phonemecorresponding to the viseme.

The viseme-generation application uses time-warping techniques on thefirst set of training data to generate a second set of training datawith audio from a second speaker, without the need to label the visemesby hand. Time warping adjusts for differences in speech of differentindividuals, such as intonation, emphasis, or speed, such that utteredphonemes in the second sequence occur at identical time stamps as thecorresponding phonemes in the first audio sequence.

More specifically, the viseme-generation application accesses a secondaudio sequence that corresponds to the sequence of phonemes generated bya second speaker speaking the same words as the first speaker. Theviseme-generation application adjusts the second audio sequence suchthat a length of the second audio sequence is equal to a length of thefirst audio sequence and such that the phonemes uttered by the secondspeaker occur at the same time stamps as the corresponding phonemesoccur in the first sequence. The timing of the phonemes is therebywarped to fit the second audio sequence.

Subsequently, the viseme-generation application matches the sequence ofvisemes to the second audio sequence. Because the viseme-generationapplication has mapped the second audio sequence to the first audiosequence, the viseme sequence corresponds to the second audio sequenceand can be reused. Hence, no hand-animation or hand-mapping of visemesis needed.

This process can continue for different speakers, and such training datacan be provided to a predictive model, thereby increasing the robustnessof the model. The viseme-generation application then trains a visemeprediction model to predict a sequence of visemes from the firsttraining set and the second training set. Optionally, theviseme-generation application represents a sequence of audio as one ormore feature vectors, provides the feature vectors to a predictive modeltrained with the training data, and obtains a prediction for a visemecorresponding to the audio sequence. Viseme-generation application canoperate in real time, thereby facilitating improved live animationsystems.

FIG. 1 is a diagram depicting a viseme-generation system, according tocertain embodiments of the present disclosure. Viseme-generation system100 includes one or more of viseme-generation application 102, audioinput device 105, Analog-to-Digital (A/D) converter 110, training data130 a-n, output viseme 135, and output device 150. Viseme-generationapplication 102 includes feature vector 115, viseme prediction model120, and visemes 140 a-n.

In an example, viseme-generation application 102 receives an audiosequence from audio input device 105, generates feature vector 115, anduses viseme prediction model 120 to a select an output viseme 135.Output viseme 135 is selected from visemes 140 a-n, each of whichcorresponds to a distinct mouth shape. Visemes are discussed furtherwith respect to FIG. 2. Feature vector 115 can include variousrepresentation of the audio sequence [and] is discussed further withrespect to FIG. 4.

Audio input device 105 can be a microphone or an analog signal thatrepresents audio. A/D converter 110 converts analog audio into digitalsamples by sampling and then quantizing the analog signal. Audio inputdevice 105 receives audio from an animator and passes the audio to A/Dconverter 110, which converts the audio into audio samples.Viseme-generation application 102 receives the audio samples from A/Dconverter 110. In an embodiment, digital audio samples are received froma network connection and passed directly into viseme-generationapplication 102. For example, the digital audio samples can be generatedvia a speech synthesis application that outputs samples representing acartoon voice based on textual or other input.

In an embodiment, viseme-generation application 102 animates a character(e.g., a computer-generated puppet) based on the determined viseme andprovides the animation to output device 150, such as a display.Alternatively, viseme-generation application 102 can provide the visemedirectly to another application such as an animation application.

In a further example, viseme-generation application 102 generatestraining data 130 a-n for training viseme prediction model 120. Morespecifically, viseme-generation application 102 uses techniques such astime-warping to generate additional sets of training data 130 b-n fromtraining data 130 a. Training data 130 a includes a first audio sequenceand a corresponding viseme sequence. Viseme-generation application 102trains viseme prediction model 120 with training data 130 a-n. Visemeprediction model 120 can be a viseme prediction model, such as arecurrent neural network or a Long Short-Term Memory (LSTM) model.

FIG. 2 depicts an example of a viseme set used by a viseme-generationsystem, according to certain embodiments of the present disclosure. FIG.2 depicts viseme set 200, which includes visemes 201-212. Each of thevisemes 201-201 corresponds to a unique mouth shape. Visemes 201-212represent, respectively, silence, a mouth appearance for an “Ah” sound,a mouth appearance for a “D” sound, a mouth appearance for a “Ee” sound,a mouth appearance for a “F” sound, a mouth appearance for a “L” sound,a mouth appearance for a “M” sound, a mouth appearance for a “Oh”, amouth appearance for a “R” sound, a mouth appearance for a “S” sound, amouth appearance for a “Uh” sound, and a mouth appearance for a “W-Oo”sound.

In some embodiments, the unique mouth shapes may not correspondaccurately with mouth shapes used by humans when speaking. For instance,the viseme may vary slightly from expected human mouth shapes due toemphasis employed in the animation, which can vary by animation style.

FIG. 3 is a flowchart that depicts an example of a process forgenerating a sequence of visemes from an audio sequence, according tocertain embodiments of the present disclosure. Process 300 is describedwith respect to viseme-generation application 102 as depicted in FIG. 1,but can be implemented by other systems.

At block 301, process 300 involves accessing speech samplescorresponding to a time period. Viseme-generation application 102 canreceive audio sequence generated in real time by audio input device 105via A/D converter 110. Viseme-generation application 102 analyzes speechsamples in sequences, or windows of time.

For example, viseme-generation application 102 can use a sliding window(e.g., 25 milliseconds) of samples with a particular stride (e.g., 10milliseconds). In this example, viseme-generation application 102buffers incoming audio samples from 0 to 25 milliseconds, creates afirst feature vector from the buffer of input samples, receives moreaudio samples and creates a second feature vector from audio samplesfrom 10 milliseconds to 35 milliseconds, and so on. A given audiosequence can include audio samples from the present, a past time period,or a future time period relative to the output viseme.

At block 302, process 300 involves computing a feature vectorrepresenting the speech samples. Viseme-generation application 102computes a feature vector 115 from the speech samples. Feature vector115 represents the sequence, or window, of audio samples in a mannersuitable for the viseme prediction model. Feature vector 115 can includedifferent features, for example, the audio samples themselves,statistics derived from the audio samples, mel-frequency cepstrumcoefficients (MFCCs) coefficients, time derivatives, energycalculations, etc. Viseme-generation application 102 can derive suchfeatures from audio samples using different methods.

FIG. 4 depicts an example of a feature vector 400 generated byviseme-generation application 102. Feature vector 400 includes MFCCcomponent 402, energy component 403, MFCC derivatives 404, and energylevel derivative 405. In an example, feature vector 115 is a28-dimensional feature vector, but other size dimensions are possible.As depicted in FIG. 4, feature vector 400 includes MFCC component 402 oflength thirteen, energy component 403 of length one, MFCC derivatives404 of length thirteen, and energy value derivative 405 of length one.

MFCCs are a frequency-based representation with non-linearly spacedfrequency bands that roughly match the response of the human auditorysystem. Feature vector 115 can include any number of MFCCs derived fromthe audio sequence.

In an embodiment, before computing MFCCs, viseme-generation application102 can filter the input audio to boost signal quality. In an example,viseme-generation application 102 compresses and boosts the input audiolevels by using a Hard Limiter filter. A Hard Limiter filter canattenuate audio that is greater in amplitude than a predefinedthreshold. The Hard Limiter filter is typically applied in conjunctionwith an input boost, which increases overall volume while avoidingdistortion.

Feature vector 115 can include energy component 403. Energy component403 represents the energy of the sequence of the audio samples in thewindow, for example, using a function such as the log mean energy of thesamples.

Feature vector 115 can include MFCC derivatives 404 or energy levelderivative 405. Including time derivatives in feature vector 115benefits viseme prediction model 120 because derivatives can makechanges in the audio more apparent to the predictive model. For example,changes in the audio can cause large changes in the MFCCs, that causethe derivatives to change, causing viseme prediction model 120 torecognize an upcoming transition between visemes in the output sequence.

Time derivatives can cause noise if computed at the same frequency asthe MFCCs. As such, viseme-generation application 102 can average thetime derivatives over a larger temporal region than the standard audiosequence window, thereby smoothing out large values.

But because such averaging of time, derivatives over multiple timewindows can cause latency due to buffering. In an embodiment, visemeprediction model 120 calculates time derivatives using averaged finitedifferences between MFCCs computed two windows before and after thecurrent MFCC window.

Returning to FIG. 3, at block 303, process 300 involves determining asequence of predicted visemes representing speech for the present subsetby applying the feature vector to the viseme prediction model. Visemeprediction model 120 is trained to predict a viseme from predeterminedvisemes. More specifically, viseme-generation application 102 providesfeature vector 115 to viseme prediction model 120. Viseme predictionmodel 120 receives a predicted output viseme 135.

Viseme prediction model 120 can be implemented with different types ofpredictive models or machine-learning models. As an example, visemeprediction model 120 can be implemented using a Long Short-Term Memory(LSTM) model.

FIG. 5 depicts an example of a LSTM neural network used byviseme-generation application, according to certain embodiments of thepresent disclosure. FIG. 5 depicts LSTM model 500, which includes delay501, inputs 501 a-n, states 509 a-n, and output visemes 510 a-n. Visemeprediction model 120 can be implemented using LSTM model 500. In thisexample, LSTM model 500 is configured as a unidirectional single-layerLSTM with a 200-dimensional hidden state that is mapped linearly to 12output viseme classes.

LSTM model 500 receives an input sequence of feature vectors a₀, a₁, a₂,. . . , a_(n) derived from sequences of streaming audio and outputs acorresponding sequence of visemes v₀, v₁, v₂, . . . v_(n). Featurevectors a₀, a₁, a₂, . . . a_(n) are indicated by inputs 501 a-n. Visemesv₀, v₁, v₂, . . . v_(n) are indicated by output visemes 910 a-c. LSTMmodel 500 includes internal states L₀, L₁, L₂, . . . L_(nn), depicted byinternal states 505 a-n. Internal states 505 a-n represent internalvalues derived from the inputs 501 a-n. Any number of internal states ispossible.

LSTM model 500 predicts visemes based on feature vectors for past,present, or future windows in time. LSTM model 500 can consider featurevectors for future windows by delaying the output of the predictedviseme until subsequent feature vectors are received and analyzed. Delay501, denoted by d, represents the number of time windows of look-ahead.For a current audio feature vector a_(t), LSTM model 500 predicts aviseme that appears d windows in the past at v_(t-d).

As depicted, the LSTM model 500 is configured with a delay of two,because two feature vectors a₀ and a₁ are processed before output visemev₀ is generated. LSTM model 500 outputs the first predicted viseme v₀that corresponds in time to the feature vector a₀, after receivingfeature vectors a₀, a₁ and a₂. As shown, feature vectors a₀, a₁ and a₂are used by LSTM model 500 in predicting output viseme v₀.

LSTM model 500 can therefore be configured with a different delay basedon particular application requirements. Determination of the delay 501involves a tradeoff between accuracy and latency. For example, a longerdelay 501 provides LSTM model 500 additional data on which to make aprediction of an output viseme 510 a, thereby improving the accuracy ofthe output viseme sequence. For example, when shortening the amount offuture audio information, output visemes may display chatter. Chatter isexcessive changing in mouth appearance reflected by the output visemeschanging too quickly. Chatter can be due in part to the fact that someanimations often change visemes slightly ahead of the speech thatcorresponds to the predicted viseme. In an embodiment, d=6 providessufficient lookahead, but adds an additional 60 milliseconds of latencyto the model.

But as discussed, in order to be realistic, animation viewed by thehuman speaker or an audience listening directly to the speaker requiresa latency below a perceptible threshold, which precludes large amountsof buffering and look-ahead. Real-time animation viewed via broadcastcan have an arbitrary delay that is not noticed by the viewers as longas audio and video signals are subjected to the same delay. But a delaythat is too long risks foreclosing real-time operation, because forreal-time systems, LSTM model 500 keeps perceptible delay below ahuman-detectable threshold. For example, in experimental results,viseme-generation application 102 can translate a 24 frames/second audiosequence into a viseme sequence with less than 200 milliseconds oflatency. Such latency is within a tolerance range for real-timeanimation, i.e., not typically perceived by a human observing theanimation.

In another embodiment, LSTM model 500 can output a probability that aparticular viseme is a match for the feature vector. For example, LSTMmodel 500 may output a probability of 72% for viseme 202 and 28% forviseme 204. In this case, the viseme-generation application 102 canselect the viseme with the highest probability, e.g., viseme 202.

In a further embodiment, viseme-generation application 102 outputs theviseme sequence at a frame rate that differs from a frame rate used foranimation. For example, viseme-generation application 102 outputsvisemes at 100 frames/second whereas animation is generated at 24frames/second. Various techniques may be used by viseme-generationapplication 102 to remove noise, or erroneous viseme artifacts createdby frame rate conversion, i.e., converting the visemes sequence from theoutput frame rate to the animation frame rate.

For example, the viseme-generation application 102 could classify aviseme as noise if that viseme is presented for less than a thresholdnumber of frames. In one example, a viseme that is displayed for oneframe is considered to be a result of frame-rate conversion noise, sinceanimators do not typically show a particular viseme for less than twoframes. To remove such noise, viseme-generation application 102 delaysoutputting the predicted viseme sequence by a predetermined number offrames. In an example, a delay of two frames is used, in accordance withsome animation practices. By delaying, viseme-generation application 102provides a look-ahead to adjust the output viseme sequence in the eventthat the output viseme is present for less than a threshold of frames.For example, viseme-generation application 102 determines that a currentframe includes a particular viseme and that neither a subsequent framenor a previous frame, e.g., a frame that is buffered, includes theparticular viseme. In response, viseme application maps the viseme ofthe previous frame to the current frame. Therefore, the output visemesequence does not have viseme transitions.

In another embodiment, viseme-generation application 102 can removenoise from the resulting viseme sequence. For example, viseme-generationapplication 102 remaps the visemes by subsampling the 100 frames/secondviseme sequence to a 24 frames/second sequence. Viseme-generationapplication 102 can determine that a particular viseme of the sequenceof visemes corresponds to one frame of video and remove the particularviseme from the sequence of visemes, replacing the removed viseme witheither the previous or the subsequent viseme.

In yet a further embodiment, in contrast to a delay implemented by LSTMmodel 500 in order to analyze future feature vectors, viseme-generationapplication 102 can create a feature vector 115 that includes MFCCinformation for a number of future windows. In this manner, informationfrom future time windows is built into a particular feature vectorrather than being separately considered by LSTM model 500.

In an embodiment for performance animation, or non-real-time use, LSTMmodel 500 can be a bi-directional LSTM. Such a configuration can be usedwhen latency is not a concern. For example, in an offline-configuration,viseme-generation application 102 can receive a set of feature vectorsderived from audio corresponding to an entire speech and operate on theentire speech simultaneously. Having feature vectors for an entiresequence, as opposed to one at a time, or a window at a time, canincrease accuracy of the predicted visemes.

Returning to FIG. 3, at block 304, process 300 involves providing avisualization corresponding to the predicted viseme by accessing a listof visualizations, mapping the viseme to a listed visualization, andconfiguring a display device to display the viseme. For example,viseme-generation application 102 accesses a list of visualizations.Each visualization in the list corresponds to a particular viseme. Forexample, viseme 205 may be animated in a certain manner that isdifferent, for example, from viseme 206. Viseme-generation application102 maps the predicted viseme to the corresponding visualization, forexample, by doing a table lookup. Viseme-generation application 102 canthen configure a display device to display the viseme.

Training the Viseme Prediction Model

As discussed, viseme prediction model 120 is trained using training data130 a-n. Training data can include a set of feature vector andcorresponding predicted visemes. Viseme-generation application 102 canbe used to generate training data 130 a-n.

Embodiments described herein use machine-learning to train visemeprediction model 120. As discussed, various types of machine-learningmodels can implement viseme prediction model 120. In a typical trainingprocess, viseme prediction model 120 learns to map sequences of inputs,typically feature vectors, to sequences of outputs. In an exampletraining process, viseme prediction model 120 learns to predict visemesfrom a diverse set of audio sequences from different speakers. As asimplified example, the training data includes a mapping between aparticular audio sequence or a particular feature vector to acorresponding output or viseme, where the feature vectors representaudio samples from different speakers. Viseme prediction model 120learns which feature vectors (and thereby which audio sequences)correspond to the particular viseme, and thereby learns to account forvariations in different parameters of the feature vectors (i.e.,variations in speaking characteristics from different speakers). Thus,with training data that includes a wide variety of audio data mapped tocorresponding visemes, trained viseme prediction model 120 canaccurately map a wide variety of speaking styles to particular visemevisualizations.

In an example, training data 130 a-n includes multiple training vectors.Each training vector includes an input sequence such as feature vectorfor an audio sequence and a corresponding output sequence such as anoutput viseme (e.g., a feature vector for the sound “Sh” and a viseme ofa mouth shape for the sound “Sh”). The corresponding output viseme for agiven audio sequence can be generated by hand, e.g., by an animator, oran automated tool such as process 600 described with respect to FIG. 6.

The sets of training data 130 a-n can be divided into a training groupand a test group. The training group of data is provided to themachine-learning model. The test group of training data is used forsubsequent testing of the trained model. In this manner, visemeprediction model 120 is not tested with the same data on which it wastrained.

FIG. 6 is a flowchart that depicts an example of a process 600 fortraining a viseme-generation system, according to certain embodiments ofthe present disclosure. Training can be an iterative process. Forexample, after viseme-generation application 102 has completed block605, process 600 can continue again with block 601 until either thetraining data set 130 a-n has been provided to the viseme predictionmodel 120, or the viseme prediction model is sufficiently trained.

Training data includes input sequences such as training vectors andcorresponding output sequences such as expected visemes for eachsequence. For example, if a particular audio sequence is of a speakeruttering the “Ah” sound, then the predicted viseme corresponds to the“Ah” sound, i.e., the mouth shape that a speaker makes when uttering thesound.

At block 601, process 600 involves determining a feature vector for eachsample of the respective audio sequence of each set of training data.For example, training data 130 a includes audio samples. In that case,the viseme-generation application 102 determines, for a window of audiosamples, feature vector 115 in a substantially similar manner asdescribed with respect to block 302 in process 300. As discussed withrespect to FIGS. 3 and 4, feature vector 115 can include one or more ofMFCC component 402, energy component 403, MFCC derivatives 404, andenergy level derivative 405.

At block 602, process 600 involves providing the feature vector to theviseme prediction model. The viseme-generation application 102 providesfeature vector 115, which represents a corresponding audio sequence, toviseme prediction model 120.

At block 603, process 600 involves receiving, from the viseme predictionmodel, a predicted viseme. The viseme-generation application 102receives a predicted viseme from viseme prediction model 120. Thepredicted viseme corresponds to the feature vector 115, and to thecorresponding input audio sequence from which the feature vector wasgenerated.

At block 604, process 600 involves calculating a loss function bycalculating a difference between predicted viseme and the expectedviseme. The expected viseme for the feature vector is included in thetraining data. The expected viseme can be generated by hand-animation,e.g., using an animator to map the audio from which the feature vectorwas generated to a viseme from the set of visemes. The loss function isused by viseme prediction model 120 to minimize error over time.

At block 605, process 600 involves adjusting internal parameters, orweights, of the viseme prediction model to minimize the loss function.With each iteration, the viseme-generation application 102 seeks tominimize the loss function until viseme prediction model 120 issufficiently trained. Viseme-generation application 102 can use abackpropagation training method to optimize internal parameters of theLSTM model 500. Backpropagation updates internal parameters of thenetwork to cause a predicted value to be closer to an expected output.Viseme-generation application 102 can use cross-entropy loss to penalizeclassification errors with respect to the expected viseme sequence. Theground truth viseme sequences can be animated at 24 frames/second andup-sampled to match the 100 frames/second frequency of the model.

Viseme-generation application 102 can continue block 601-605 of process600 as necessary until viseme prediction model 120 is sufficientlytrained. At a point at which adequate training has been performed, theviseme-generation application 102 can test the viseme prediction model.For each test vector, the application provides the corresponding featurevector to the viseme prediction model 120. The viseme-generationapplication 102 receives a predicted viseme from the viseme predictionmodel 120.

The predicted viseme and the expected viseme can be compared indifferent ways. For example, an automated system can be used.Alternatively, a training data generation system can provide a displaythat shows a user the predicted viseme sequence and an expected visemesequence. The user can indicate which sequences are more realistic oraccurate by providing feedback to the training data generation system.

If viseme-generation application 102 determines that the visemeprediction model 120 is predicting incorrect visemes for a thresholdnumber of instances, then viseme prediction model 120 can provideadditional training data 130 a-n to the viseme prediction model 120 andre-test accordingly.

As discussed, training data can involve using human animators to mapaudio sequences to predicted visemes. Such a process, while useful, canbe expensive in time and cost. Because a threshold amount of trainingdata is needed such that viseme prediction model 120 is sufficientlytrained, generating training data by hand can make the use of suchmodels impractical.

Embodiments described herein use automatic speech alignment such astime-warping techniques to generate, from a first set of training data,additional sets of training data for different speakers. Morespecifically, viseme-generation application 102 can automaticallypropagate hand animated visemes for a first audio sequence spoken by afirst speaker to a second audio sequence spoken by a second speaker. Inso doing, viseme-generation application 102 removes the need for thesecond speaker's speech to be hand-animated as was done to the firstspeaker's speech. Embodiments can increase the amount of availabletraining data by a factor of four or more and can produce acceptableresults with as little as thirteen to nineteen minutes of hand-authoredlip sync data.

FIG. 7 is a flowchart that depicts an example of a process forgenerating training data, according to certain embodiments of thepresent disclosure. FIG. 7 is shown in conjunction with FIG. 8. FIG. 8depicts an example of time-warping used to generate training data,according to certain embodiments of the present disclosure. FIG. 8includes training data 800, which includes a first set of training data801 and a second set of training data 802. First set of training data801 includes viseme sequence 811 and first audio sequence 812. Secondset of training data 802 includes adjusted audio sequence 814 and visemesequence 815.

At block 701, process 700 involves accessing a first set of trainingdata including a first audio sequence representing a sentence spoken bya first speaker and having a first length. For example,viseme-generation application 102 accesses the first set of trainingdata 801. The first set of training data 801 includes viseme sequence811 and first audio sequence 812.

The audio samples in first audio sequence 812 represent a sequence ofphonemes. The visemes in viseme sequence 811 are a sequence of visemes,each of which correspond to one or more audio samples in first audiosequence 812. Viseme sequence 811 can be hand-generated. For example, ananimator lip syncs sentences from a particular dataset. The first set oftraining data can be training data 130a.

At block 702, process 700 involves accessing a second audio sequencerepresenting the sentence spoken by a second speaker and having a secondlength. Second audio sequence 813 includes the sequence of phonemes.Viseme-generation application 102 warps a second recording of the samesentence as spoken in the first sequence to match the timing of thesecond speaker to the first speaker. In this manner, viseme-generationapplication 102 can reuse the same viseme sequence 811 with multipledifferent input streams from multiple different speakers.

At block 703, process 700 involves adjusting the second audio sequencesuch that (i) a second sequence length is equal to the first length and(ii) at least one phoneme occurs at the same time stamp in the firstsequence and in the second sequence, thereby creating a second set oftraining data. Viseme-generation application 102 adjusts the secondaudio sequence 813 to match the first audio sequence 812, therebycreating adjusted audio sequence 814.

Viseme-generation application 102 maps the second sequence to the firstsequence such that the sounds or phonemes within the audio sequenceoccur at the same time in each sequence. In an example, the first audiosequence reflects the first speaker speaking the sound “Ah” at aparticular time stamp. The second speaker most likely did not speak thesound “Ah” at precisely the same time as the first speaker did.Therefore, viseme-generation application 102 maps the second audiosequence to the first audio sequence such that the corresponding sound“Ah” occurs at the same time stamp.

Because different speakers emphasize different sounds or phonemes, andspeak at different speeds, the adjustment of the second audio sequenceis non-linear. For example, the time adjustment made to a particularphoneme may be different than an adjustment made for another phoneme.Similarly, a section of the second audio sequence relative to thecorresponding part of the first audio sequence may be compressed inlength, whereas a sequence spoken more quickly than the first may beexpanded.

At block 704, process 700 involves mapping the sequence of visemes tothe second audio sequence. Viseme-generation application 102 adjusts thesecond audio sequence such that a length of the second audio sequence isequal to the length of the first audio sequence and such that thephonemes uttered by the second speaker occur at the same time stamps asthe corresponding phonemes occur in the first sequence. In this manner,the timing of the phonemes is thereby warped to fit the second audiosequence. With the second audio sequence mapped to the first audiosequence, the viseme sequence, which corresponds to the first audiosequence, also now corresponds to the second audio sequence. By sodoing, the viseme-generation application 102 has created a second set oftraining data that includes adjusted audio sequence 814 and visemesequence 815.

At block 705, process 700 involves training a viseme prediction model topredict a sequence of visemes from the first training set and the secondtraining set. Training occurs in a substantially similar fashion asdescribed in process 600.

In an embodiment, viseme-generation application 102 can warp both afirst audio sequence and a corresponding sequence of visemes to a secondaudio sequence, rather than warping the second audio sequence that lacksa corresponding set of visemes to a first audio sequence, as describedwith respect to process 700.

For example, viseme-generation application 102 receives a first set oftraining data including a first audio sequence and a corresponding setof visemes and a second set of training data including a second audiosequence. Viseme-generation application 102 adjusts the first audiosequence such that a length of the first sequence is equal to a lengthof the second sequence and warps the set of visemes to match the secondsequence, thereby creating a second set of training data.

Because the second audio sequence is unmodified, viseme-generationapplication 102 preserves more natural variations in the voice of thesecond audio sequence, as opposed to training viseme prediction model120 with the warped second sequence. Viseme-generation application 102provides an unmodified version of the first set of training data or thesecond set of training data to viseme prediction model 120.

In this manner, viseme-generation application 102 trains visemeprediction model 120 with two sets of training data, each of whichcontains audio that is unmodified. In contrast, process 700, includes asecond audio sequence that is modified from its original form.

Example of a Computing System for Implementing Certain Embodiments

Any suitable computing system or group of computing systems can be usedfor performing the operations described herein. For example, FIG. 9depicts an example of a computing system for implementing certainembodiments of the present disclosure. The implementation of computingsystem 900 could be used for one or more of viseme-generationapplication 102 or viseme predictive model 120.

The depicted example of a computing system 900 includes a processor 902communicatively coupled to one or more memory devices 904. The processor902 executes computer-executable program code stored in a memory device904, accesses information stored in the memory device 904, or both.Examples of the processor 902 include a microprocessor, anapplication-specific integrated circuit (“ASIC”), a field-programmablegate array (“FPGA”), or any other suitable processing device. Theprocessor 902 can include any number of processing devices, including asingle processing device.

A memory device 904 includes any suitable non-transitorycomputer-readable medium for storing program code 905, program data 907,or both. Program code 905 and program data 907 can be fromviseme-generation application 102, viseme prediction model 120, or anyother applications or data described herein. A computer-readable mediumcan include any electronic, optical, magnetic, or other storage devicecapable of providing a processor with computer-readable instructions orother program code. Non-limiting examples of a computer-readable mediuminclude a magnetic disk, a memory chip, a ROM, a RAM, an ASIC, opticalstorage, magnetic tape or other magnetic storage, or any other mediumfrom which a processing device can read instructions. The instructionsmay include processor-specific instructions generated by a compiler oran interpreter from code written in any suitable computer-programminglanguage, including, for example, C, C++, C#, Visual Basic, Java,Python, Perl, JavaScript, and ActionScript.

The computing system 900 may also include a number of external orinternal devices, an input device 920, a presentation device 918, orother input or output devices. For example, the computing system 900 isshown with one or more input/output (“I/O”) interfaces 908. An I/Ointerface 908 can receive input from input devices or provide output tooutput devices. One or more buses 906 are also included in the computingsystem 900. The bus 906 communicatively couples one or more componentsof a respective one of the computing system 900.

The computing system 900 executes program code 905 that configures theprocessor 902 to perform one or more of the operations described herein.Examples of the program code 905 include, in various embodiments,modeling algorithms executed by the viseme-generation application 102,or other suitable applications that perform one or more operationsdescribed herein. The program code may be resident in the memory device904 or any suitable computer-readable medium and may be executed by theprocessor 902 or any other suitable processor.

In some embodiments, one or more memory devices 904 stores program data907 that includes one or more datasets and models described herein.Examples of these datasets include interaction data, environmentmetrics, training interaction data or historical interaction data,transition importance data, etc. In some embodiments, one or more ofdata sets, models, and functions are stored in the same memory device(e.g., one of the memory devices 904). In additional or alternativeembodiments, one or more of the programs, data sets, models, andfunctions described herein are stored in different memory devices 904accessible via a data network.

In some embodiments, the computing system 900 also includes a networkinterface device 910. The network interface device 910 includes anydevice or group of devices suitable for establishing a wired or wirelessdata connection to one or more data networks. Non-limiting examples ofthe network interface device 910 include an Ethernet network adapter, amodem, and/or the like. The computing system 900 is able to communicatewith one or more other computing devices via a data network using thenetwork interface device 910.

In some embodiments, the computing system 900 also includes the inputdevice 920 and the presentation device 918 depicted in FIG. 9. An inputdevice 920 can include any device or group of devices suitable forreceiving visual, auditory, or other suitable input that controls oraffects the operations of the processor 902. Non-limiting examples ofthe input device 920 include a touchscreen, a mouse, a keyboard, amicrophone, a separate mobile computing device, etc. A presentationdevice 918 can include any device or group of devices suitable forproviding visual, auditory, or other suitable sensory output.Non-limiting examples of the presentation device 918 include atouchscreen, a monitor, a speaker, a separate mobile computing device,etc. Presentation device 918 is configurable to display animationsderived from an output sequence of visemes. In addition, presentationdevice 918 can display user interface elements, such as sliders orcontrols, that allow configuration of combined parameters 140.

Although FIG. 9 depicts the input device 920 and the presentation device918 as being local to the computing device that executesviseme-generation application 102, other implementations are possible.For instance, in some embodiments, one or more of the input device 920and the presentation device 918 can include a remote client-computingdevice that communicates with the computing system 900 via the networkinterface device 910 using one or more data networks described herein.

General Considerations

Numerous specific details are set forth herein to provide a thoroughunderstanding of the claimed subject matter. However, those skilled inthe art will understand that the claimed subject matter may be practicedwithout these specific details. In other instances, methods,apparatuses, or systems that would be known by one of ordinary skillhave not been described in detail so as not to obscure claimed subjectmatter.

Unless specifically stated otherwise, it is appreciated that throughoutthis specification discussions utilizing terms such as “processing,”“computing,” “calculating,” “determining,” and “identifying” or the likerefer to actions or processes of a computing device, such as one or morecomputers or a similar electronic computing device or devices, thatmanipulate or transform data represented as physical electronic ormagnetic quantities within memories, registers, or other informationstorage devices, transmission devices, or display devices of thecomputing platform.

The system or systems discussed herein are not limited to any particularhardware architecture or configuration. A computing device can includeany suitable arrangement of components that provide a result conditionedon one or more inputs. Suitable computing devices include multi-purposemicroprocessor-based computer systems accessing stored software thatprograms or configures the computing system from a general purposecomputing apparatus to a specialized computing apparatus implementingone or more embodiments of the present subject matter. Any suitableprogramming, scripting, or other type of language or combinations oflanguages may be used to implement the teachings contained herein insoftware to be used in programming or configuring a computing device.

Embodiments of the methods disclosed herein may be performed in theoperation of such computing devices. The order of the blocks presentedin the examples above can be varied—for example, blocks can bere-ordered, combined, and/or broken into sub-blocks. Certain blocks orprocesses can be performed in parallel.

The use of “adapted to” or “configured to” herein is meant as open andinclusive language that does not foreclose devices adapted to orconfigured to perform additional tasks or steps. Additionally, the useof “based on” is meant to be open and inclusive, in that a process,step, calculation, or other action “based on” one or more recitedconditions or values may, in practice, be based on additional conditionsor values beyond those recited. Headings, lists, and numbering includedherein are for ease of explanation only and are not meant to belimiting.

While the present subject matter has been described in detail withrespect to specific embodiments thereof, it will be appreciated thatthose skilled in the art, upon attaining an understanding of theforegoing, may readily produce alterations to, variations of, andequivalents to such embodiments. Accordingly, it should be understoodthat the present disclosure has been presented for purposes poses ofexample rather than limitation, and does not preclude the inclusion ofsuch modifications, variations, and/or additions to the present subjectmatter as would be readily apparent to one of ordinary skill in the art.

1. A method of predicting visemes from an audio sequence, the methodcomprising: accessing a first set of training data comprising: (i) afirst audio sequence of samples representing a sentence spoken by afirst speaker and having a first length, wherein the audio sequencerepresents a sequence of phonemes, and (ii) a sequence of visemes,wherein each viseme is mapped to a respective audio sample of the firstaudio sequence; creating a second set of training data by: accessing asecond audio sequence of samples representing the same sentence spokenby a second speaker and having a second length, wherein the second audiosequence comprises the sequence of phonemes; adjusting the second audiosequence such that (i) a second sequence length is equal to the firstlength and (ii) at least one phoneme occurs at the same time stamp inthe first audio sequence and in the second audio sequence; mapping thesequence of visemes to the second audio sequence; and training a visemeprediction model to predict a sequence of visemes from the first set oftraining data and the second set of training data.
 2. The method ofclaim 1, wherein training the viseme prediction model comprises:determining a feature vector for each sample of the respective audiosequence of each set of training data; providing the feature vectors tothe viseme prediction model; receiving, from the viseme predictionmodel, a predicted viseme; calculating a loss function by calculating adifference between the predicted viseme and an expected viseme; andadjusting internal parameters of the viseme prediction model to minimizethe loss function.
 3. The method of claim 2, wherein the feature vectorcomprises: a set of mel-frequency cepstrum coefficients for theplurality of speech samples, a logarithm of a mean energy of theplurality of speech samples, and a first temporal derivative of theplurality of speech samples.
 4. The method of claim 1, furthercomprising: accessing a plurality of speech samples corresponding to atime period, wherein a present subset of the speech samples correspondsto a present time period and a past subset of the speech samplescorresponds to a past time period; computing a feature vectorrepresenting the plurality of speech samples; determining a sequence ofpredicted visemes representing speech for the present subset by applyingthe feature vector to the viseme prediction model trained to predict aviseme from a plurality of predetermined visemes, wherein the predictionis based on the past subset and the present subset; and providing avisualization corresponding to the sequence of predicted visemes,wherein providing the visualization comprises: accessing a list ofvisualizations, mapping the viseme to a listed visualization, andconfiguring a display device to display the listed visualization.
 5. Themethod of claim 4, further comprising: mapping each of the sequence ofvisemes to a frame rate; determining that a particular viseme of thesequence of visemes corresponds to a frame of video; and removing theparticular viseme from the sequence of visemes.
 6. The method of claim4, further comprising: mapping each of the sequence of visemes to aframe rate; delaying an output of the sequence of predicted visemes by apredetermined number of frames; and responsive to determining that (i) acurrent frame includes a particular viseme and (ii) a subsequent frameand a previous frame lack the particular viseme, mapping the viseme ofthe previous frame to the current frame.
 7. The method of claim 4,further comprising: mapping each of the sequence of visemes to a framerate; and representing the sequence of visemes on a graphical timelineaccording to the frame rate.
 8. A system comprising: a computer-readablemedium storing non-transitory computer-executable program instructionsfor applying an image effect within an image processing application; anda processing device communicatively coupled to the computer-readablemedium for executing the non-transitory computer-executable programinstructions, wherein executing the non-transitory computer-executableprogram instructions configures the processing device to performoperations comprising: accessing a plurality of speech samplescorresponding to a time period, wherein a present subset of the speechsamples corresponds to a present time period and a past subset of thespeech samples corresponds to a past time period; computing a featurevector representing the plurality of speech samples; determining asequence of predicted visemes representing speech for the present subsetby applying the feature vector to a viseme prediction model trained witha second training data set comprising a second audio sequence spoken bya second speaker and a sequence of visemes, wherein the second trainingdata set is created by mapping the second audio sequence to a firstaudio sequence; and providing a visualization corresponding to thesequence of predicted visemes, wherein providing the visualizationcomprises: accessing a list of visualizations, mapping the viseme to alisted visualization, and configuring a display device to display thelisted visualization.
 9. The system of claim 8, further comprising:increasing an amplitude of each of the plurality of speech samples;determining, from the plurality of speech samples, a speech sample thathas an amplitude greater than a threshold; and reducing the amplitude ofthe speech sample.
 10. The system of claim 8, wherein computing thefeature vector further comprises: calculating a set of mel-frequencycepstrum coefficients for the plurality of speech samples, calculating alogarithm of a mean energy of the plurality of speech samples, andcalculating a first temporal derivative of the plurality of speechsamples.
 11. The system of claim 8, the operations further comprising:mapping each of the sequence of visemes to a frame rate; delaying anoutput of the sequence of predicted visemes by a predetermined number offrames; and responsive to determining that (i) a current frame includesa particular viseme and (ii) a subsequent frame and a previous framelack the particular viseme, mapping the viseme of the previous frame tothe current frame.
 12. The system of claim 8, the operations furthercomprising: mapping the sequence of predicted visemes to a frame rate;and representing the sequence of predicted visemes on a graphicaltimeline according to the frame rate.
 13. A computer-readable storagemedium storing non-transitory computer-executable program instructions,wherein when executed by a processing device, the program instructionscause the processing device to perform operations comprising: accessinga first set of training data comprising: (i) a first audio sequencerepresenting a sentence spoken by a first speaker and having a firstlength, wherein the first audio sequence represents a sequence ofphonemes and has a first length, and (ii) a sequence of visemes, whereineach viseme is mapped to a respective audio sample of the first audiosequence; creating a second set of training data by: accessing a secondaudio sequence representing the sentence spoken by a second speaker andhaving a second length, wherein the second audio sequence comprises thesequence of phonemes; adjusting the first audio sequence such that (i)the first length is equal to the second length and (ii) at least onephoneme occurs at the same time stamp in the first audio sequence and inthe second audio sequence; adjusting the sequence of visemes to theadjusted first audio sequence; and training a viseme prediction model topredict a sequence of visemes from the first set of training data andthe second set of training data.
 14. The computer-readable storagemedium of claim 13, wherein training the viseme prediction modelcomprises: determining a feature vector for each sample of therespective audio sequence of each set of training data; providing thefeature vectors to the viseme prediction model; receiving, from theviseme prediction model, a predicted viseme; calculating a loss functionby calculating a difference between the predicted viseme and an expectedviseme; and adjusting internal parameters of the viseme prediction modelto minimize the loss function.
 15. The computer-readable storage mediumof claim 14, wherein the feature vector comprises: a set ofmel-frequency cepstrum coefficients for each speech sample, a logarithmof a mean energy of each speech sample, and a first temporal derivativeof each speech sample.
 16. The computer-readable storage medium of claim13, wherein program instructions further cause the processing device toperform operations comprising: accessing a plurality of speech samplescorresponding to a time period, wherein a present subset of the speechsamples corresponds to a present time period and a past subset of thespeech samples corresponds to a past time period; computing a featurevector representing the plurality of speech samples; determining asequence of predicted visemes representing speech for the present subsetby applying the feature vector to the viseme prediction model trained topredict a viseme from a plurality of predetermined visemes, wherein theprediction is based on the past subset and the present subset; andproviding a visualization corresponding to the sequence of predictedvisemes, wherein providing the visualization comprises: accessing a listof visualizations, mapping the viseme to a listed visualization, andconfiguring a display device to display the listed visualization. 17.The computer-readable storage medium of claim 16, further comprising:mapping each of the sequence of visemes to a frame rate; determiningthat a particular viseme of the sequence of visemes corresponds to aframe of video; and removing the particular viseme from the sequence ofvisemes.
 18. The computer-readable storage medium of claim 16, whereinprogram instructions further cause the processing device to performoperations comprising: mapping each of the sequence of visemes to aframe rate; delaying an output of the sequence of predicted visemes by apredetermined number of frames; and responsive to determining that (i) acurrent frame includes a particular viseme and (ii) a subsequent frameand a previous frame lack the particular viseme, mapping the viseme ofthe previous frame to the current frame.
 19. The computer-readablestorage medium of claim 16, wherein program instructions further causethe processing device to perform operations comprising: mapping thesequence of predicted visemes to a frame rate; and representing thesequence of predicted visemes on a graphical timeline according to theframe rate.
 20. The computer-readable storage medium of claim 16,further comprising: increasing an amplitude of each of the plurality ofspeech samples; determining, from the plurality of speech samples, aspeech sample that has an amplitude greater than a threshold; andreducing the amplitude of the speech sample.